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Self-Regulated Strategy 


Self-Regulated Strategy Development (SRSD) is an intervention 
designed to improve students’ academic skills through a six-step 
process that teaches students specific academic strategies and self¬ 
regulation skills. The practice is especially appropriate for students 
with learning disabilities, the focal population of the current report. 

The intervention begins with teacher direction and ends with students 
independently applying the strategy, such as planning and organizing 
ideas before writing an essay. More specifically, the six steps involve 
the teacher providing background knowledge, discussing the strategy 
with the student, modeling the strategy, helping the student memorize 
the strategy, supporting the strategy, and then watching as the student 
independently performs the strategy. A key part of the process is teach¬ 
ing self-regulation skills, such as goal-setting and self-monitoring, which 
aim to help students apply the strategy without guidance. The steps can 
be combined, changed, reordered, or repeated, depending on the needs 
of the student. The SRSD model can be used with students in grades 2 
through 12 in individual, small group, or whole classroom settings. 

Research 2 

The What Works Clearinghouse (WWC) identified 10 studies of SRSD that both fall within the scope of the Students 
With a Specific Learning Disability topic area and meet WWC pilot single-case design standards. No studies meet 
WWC group design standards. No studies meet pilot single-case design standards without reservations, and 10 
studies meet pilot single-case design standards with reservations. Together, these studies included 43 children 
ages 7 to 16 who had a specific learning disability. 
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This intervention report presents findings 
from a systematic review of SRSD 
conducted using the WWC Procedures and 
Standards Handbook (version 3.0) and the 
Students With a Specific Learning Disability 
review protocol (version 3.0). 


Threshold to include single-case design evidence in WWC effectiveness ratings 


All single-case design experiments presented in the same research article are considered as one study. Results from single¬ 
case design studies contribute to the WWC effectiveness rating for an outcome domain only if the studies with outcomes in that 
domain meet a set of threshold criteria, reflecting replication across different studies, research teams, and cases. 

Specifically, these criteria include the following: (1) at least five studies that examine the intervention must meet WWC pilot 
single-case design standards without reservations or meet WWC pilot single-case design standards with reservations; (2) the 
single-case design studies must be conducted by at least three different research teams with no overlapping authorship at three 
different institutions; and (3) the combined number of cases (that is, participants or classrooms) must total at least 20. 

For more information, please refer to the Pilot Single-Case Design Standards in Appendix E of the WWC Procedures and 
Standards Handbook (version 3.0). 
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The results from single-case design studies affect the WWC effectiveness rating for an outcome domain only if the studies 
with outcomes in that domain collectively meet a set of threshold criteria. (The text box above describes these criteria.) 

The evidence from single-case design studies of SRSD on students with a specific learning disability reaches the 
required threshold to include single-case design evidence in the effectiveness ratings for one outcome domain — 
writing achievement. The evidence from the single-case design studies for SRSD does not reach the threshold to 
include single-case design evidence in the effectiveness ratings for one outcome domain —math achievement. 3 No 
studies meet standards in the 10 other domains, so this intervention report does not report on the effectiveness of 
SRSD for those domains. 4 (See the Effectiveness Summary on p. 6 for further description of all domains.) 

Effectiveness 

Based on evidence from single-case design studies, SRSD had potentially positive effects on writing achieve¬ 
ment for students with a specific learning disability. The evidence from the single-case design studies for SRSD 
does not reach the threshold to include single-case design evidence in the effectiveness ratings for the math 
achievement domain. 


Table 1. Summary of findings from single-case studies 5 


Outcome domain 

Number of 
studies 

Number of 
research 
teams 

Number of 
cases 3 

Rating of 
effectiveness 

Percentage of 

SCD experiments 
demonstrating a 
positive effect (#) 

Percentage of 

SCD experiments 
demonstrating a 
negative effect (#) 

Writing achievement 

9 

3 

37 

Potentially positive 
effects 

88% 

(15/17) 

0% 

(0/17) 

Math achievement 

1 

1 

3 

na 

na 

na 


Table Notes: In single-case design research, a case, such as a student or classroom, is the unit of intervention administration and data analysis. An experiment examines a single 
outcome measure repeatedly within and across different phases defined by the presence or absence of the intervention. There might be multiple experiments for a case if more 
than one outcome is examined, for example. All experiments within a research article are considered as one single-case design study. For the math achievement domain, the 
rating of effectiveness and percentage of single-case design experiments demonstrating a positive or negative effect are not applicable (na) because the studies with outcomes in 
this domain do not meet the threshold criteria to include single-case design evidence in the effectiveness ratings. SCD = single-case design. 

a In this intervention report, three cases represented classrooms with two students each. The rest of the cases were single students. Therefore, the writing achievement domain 
included a total of 40 students. 


Self-Regulated Strategy Development November 2017 


Page 2 






WWC Intervention Report 


Intervention Information 

Background 

SRSD is an academic practice developed by Karen Harris and Steven Graham in the early 1980s for teaching 
writing and other related skills to young students with learning difficulties. The practice has evolved over time 
and can now be used with students in grades 2 through 12 across multiple content areas, and in individual, small 
group, or whole classroom settings. Although SRSD can be used with all students, it is especially appropriate for 
students with learning disabilities, the focal population of the current report. Students learn specific strategies 
for carrying out tasks, such as writing essays or completing math problems, and learn procedures for regulat¬ 
ing the use of the strategies they have learned, such as goal-setting and self-monitoring. Free SRSD materials 
are available through thinkSRSD (www.thinksrsd.com), an online resource developed to help school districts and 
educators implement SRSD in the classroom. Lesson plans for many of the SRSD strategies are also available 
in Harris, K. R., Graham, S., Mason, L., and Friedlander, B. (2008). Powerful writing strategies for all students. 
Baltimore, MD: Brookes. 

Intervention details 

SRSD is an academic practice that involves teachers’ use of explicit instruction to teach students concrete strate¬ 
gies to improve writing and other academic skills. Strategies vary depending on the content area as well as the 
age and needs of the student. For example, teachers might introduce individual students to a specific strategy that 
will help them plan and brainstorm ideas for writing a story. Teachers often provide the students with mnemonic 
devices to ensure they remember to include all required steps or elements when carrying out the strategy. Teachers 
instruct students on how to monitor and evaluate their use of the strategy through self-regulation, and then model 
both the use of the strategy and self-regulation skills. They then ensure that students have memorized the strategy 
and can apply it correctly on their own. 

Lessons generally occur at least three times a week, and usually last 20 to 60 minutes. SRSD instruction typi¬ 
cally includes the following six stages that can be reordered, combined, changed, and repeated, depending on 
students’ needs: 

1. Develop background knowledge. The teacher identifies skill deficits and helps students develop the prerequisite 
skills needed to understand, learn, and apply the strategy. 

2. Discuss the strategy. The teacher and students discuss the students’ performance to identify areas for improve¬ 
ment and help motivate the students to use the strategy. Students might also identify one or more areas in which 
they seek to improve and learn how to monitor progress. The teacher introduces the strategies and the rationale 
for each step, often using a mnemonic device to help students remember each component. 

3. Model the strategy. The teacher models the strategy as many times as necessary, using a think-aloud process 
and voicing positive self-statements. The teacher and students discuss the advantages and challenges of the 
strategy and think about possible ways to improve the strategy. Teachers often introduce goal-setting concepts, 
and students can set individual targets to improve on baseline performance. 

4. Memorize the strategy. Students use mnemonic devices to memorize the steps required to apply the strategy. 
Students can paraphrase steps, and the teacher can use prompts for students who struggle. 

5. Support the strategy. Students gradually take responsibility for applying the strategy. The teacher provides sup¬ 
port that is tailored to the needs of the student, provides frequent constructive feedback, and offers positive 
reinforcement. Teachers can also engage other classmates by organizing peer groups to help promote strategy 
use outside of SRSD instruction. For example, classmates within a peer group can share how they make deci¬ 
sions while writing. 
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6. Independent performance. Students consistently and effectively use a strategy, often in multiple settings and 
with different tasks. Students learn how using the strategy improves their performance and how to modify the 
strategy as appropriate. 

During the instructional process, the teacher gradually transfers responsibility for implementing the strategy to 
students, until the students can adequately perform the tasks without support. After the SRSD training period has 
ended, students demonstrate how to use the skills they learned on their own. Teachers and researchers measure 
the impact of the SRSD intervention during this post-training phase, after implementing the full SRSD intervention. 
They often also measure the long-term impacts of SRSD during later maintenance phases to determine whether 
effects were sustained over time. 


Cost 

SRSD is a set of practices, not a product that can be purchased. An online resource, thinkSRSD (www.thinksrsd. 
com), provides free materials designed to help school districts and educators implement SRSD in the classroom. In 
addition, as of November 2017, professional development on the use of SRSD is available through thinkSRSD and 
ranges in price from $100 for online courses to $495 for in-person courses. Additional information about school¬ 
wide costs of teacher training or implementation of SRSD is not available. 
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Research Summary 


The WWC identified two eligible group design studies and 14 eligible 
single-case design studies that investigated the effects of SRSD on 
students with a specific learning disability. An additional 115 studies 
were identified but do not meet WWC eligibility criteria (see the Glos¬ 
sary of Terms in this document for a definition of this term and other 
commonly used research terms) for review in this topic area. Citations 
for all 131 studies are in the References section, which begins on p. 7. 


Table 2. Scope of reviewed research 


Grades 

2-10 

Delivery method 

Individual 

Intervention type 

Practice 


The WWC reviewed the two group design studies against group design standards. No studies meet group design 
standards with or without reservations. 


The WWC reviewed the 14 single-case design studies against pilot single-case design standards. No studies meet 
pilot single-case design standards without reservations, and 10 studies meet pilot single-case design standards 
with reservations. This report summarizes those 10 studies. The remaining four studies do not meet pilot single¬ 
case design standards. 

Nine of the 10 studies that meet pilot single-case design standards with reservations have at least one outcome 
in a domain that reaches the threshold for including single-case design evidence in the effectiveness ratings. 
Appendices A-C describe the details of these studies. The remaining study that meets WWC pilot single-case 
design standards with reservations has outcomes only in a domain that does not reach the threshold for includ¬ 
ing single-case design evidence in the effectiveness ratings in this report; more details on this study are available 
in Appendix D. 6 


Summary of studies meeting WWC pilot single-case design standards without reservations 

No studies of SRSD meet WWC single-case design standards without reservations. 

Summary of studies meeting WWC pilot single-case design standards with reservations 

Ten studies have experiments that meet WWC pilot single-case design standards with reservations. 7 These experi¬ 
ments investigated the effects of SRSD on writing and math achievement outcomes. The experiments included 
students ages 7 to 16 with a specific learning disability. Appendices A-D describe the details of these studies. 
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Effectiveness Summary 

The WWC review of SRSD for the Students With a Specific Learning Disability topic area includes student out¬ 
comes in 12 domains: alphabetics, comprehension, general reading achievement, math achievement, problem 
behavior, reading fluency, school engagement, science achievement, self-determination, social-emotional com¬ 
petence, social studies achievement, and writing achievement. The 10 studies of SRSD that met WWC pilot 
single-case design standards reported findings in two of the 12 domains: (a) writing achievement and (b) math 
achievement. Effectiveness ratings of SRSD for the Students With a Specific Learning Disability topic area are 
presented for one of the two domains (writing achievement). The findings from the math achievement domain do 
not meet the threshold to include single-case design evidence in the effectiveness ratings in this report. 8 For a more 
detailed description of the rating of effectiveness for single-case design studies and extent of evidence criteria, see 
the WWC Rating Criteria that starts on p. 39. 


Summary of effectiveness for the writing achievement domain 

Table 3. Rating of effectiveness for single-case design studies for the writing achievement domain 


Rating of effectiveness 

Criteria met 

Potentially positive effects 

Strong evidence of a positive 
effect with no overriding contrary 
evidence. 

Across the 17 single-case design experiments for the writing achievement domain, 15 experiments (88%) 
documented a positive effect, and 0 experiments documented a negative effect. All 15 of these single-case 
design experiments met WWC pilot SCD standards with reservations; because no studies met WWC pilot SCD 
standards without reservations, the rating of effectiveness is potentially positive effects. 


Nine studies that met WWC pilot single-case design standards with reservations reported findings in the writ¬ 
ing achievement domain. Author-reported findings for each study are available in Appendix A. The results of the 
WWC’s visual analysis of each single-case design experiment are available in Appendix C. Across the 17 single¬ 
case design experiments included in these nine studies, 15 experiments (88%) documented a positive effect, and 
0 experiments documented a negative effect. 9 This results in a rating of potentially positive effects for the writing 
achievement domain. 
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ed.gov/?id=EJ991683 This study is ineligible for review because it does not use a sample aligned with the 
protocol. 
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Appendix A.1: Research details for Danoff et al. (1993) 

Danoff, B., Harris, K. R., & Graham, S. (1993). Incorporating strategy instruction within the writing pro¬ 
cess in the regular classroom: Effects on the writing of students with and without learning disabili 
ties. Journal of Reading Behavior, 25(3), 295-322. 


Setting 

The study was conducted in a suburban elementary school located in the northeastern United 


States. The intervention took place in three general education classrooms, including two fifth- 
grade classrooms and one fourth-grade classroom, during a period called “writers’ work¬ 
shop.” The school’s special education teacher took primary responsibility for developing and 
delivering the lessons included in this study, with back-up provided by the general education 
classroom teachers. The school had about 370 students, and there were 25 to 28 students in 

each class. 

Study sample 

The study sample included two fifth-grade students (one male, one female) and one fourth-grade 
student (female) who were identified as having learning disabilities (LD) by their school district. 
Each student had an IQ above 85, with achievement at least one standard deviation below grade 
level. The fifth graders were White, and the fourth grader was Asian. The study also included 
experiments for three additional participants who did not have LD. The experiments for these 
students are not described in this report or included in the ratings of effectiveness. 

Intervention 

The Self-Regulated Strategy Development (SRSD) intervention model was used to teach 
students how to improve their writing skills. The writing and self-regulation strategies used 
in this study were taught to all students through a series of mini-lessons delivered by the 
school’s special education teacher. During the SRSD training phase, the teacher introduced a 
writing strategy to the class and gave them a small chart and a mnemonic device for remem¬ 
bering the seven basic parts of a story. The mnemonic device was ““WWW, What=2, How=2” 
and asked students to think about the following prompts: “Who are the main characters? 

When does the story take place? Where does the story take place? What do the main char¬ 
acters want to do? What happens when the main characters try to do it? How does the story 
end? How do the main characters feel?” Progression through the stages of instruction was 
criterion-based rather than time-based. The students required a series of nine to 11 mini¬ 
lessons to master the strategy and self-regulation procedures. The post-training story probes 
were administered immediately following SRSD instruction. 

Comparison 

The study used a multiple baseline design across students for each outcome. During the base¬ 
line condition for each student, teachers taught their classes as usual, and the special educa¬ 
tion teacher provided instructional assistance to students with disabilities. 

Outcomes and 
measurement 

The outcomes were the number of story elements and the story grammar scale score, which 
both fall within the writing achievement domain. For a more detailed description of these 
outcome measures, see Appendix B. The study also examined the number of words written 
and the holistic rating scale; these outcomes did not meet standards because they were not 
displayed graphically. In addition, the authors measured strategy usage and self-efficacy, but 
these outcomes are not eligible under the study review protocol. Results from the two experi¬ 
ments with outcomes in the writing achievement domain are presented in Appendix C.1. 
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Support for 
implementation 

The special education teacher adapted lesson plans from previous studies of SRSD and fol¬ 
lowed all steps of the SRSD model. 

Author-reported 

findings 

The study authors found that SRSD had a positive impact on the story grammar scale score 
and the number of story elements for all students. The results of WWC’s corresponding visual 
analysis are presented in Appendix C. 

Maintenance 

The students’ general education classroom teacher for the writers’ workshop administered a 
generalization story probe immediately following the post-training story probes. In addition, 
maintenance story probes were administered 2 weeks and 4 weeks following the comple¬ 
tion of instruction. In general, the students’ scores during the maintenance and generalization 
probes were similar to their post-training phases. However, there was a downward trend for 
the story grammar scale score. 

WWC study rating 

Meets WWC Pilot Single-Case Design Standards With Reservations. 

Reason for study rating: The experiments for both outcomes used multiple probe designs 
across participants. The baseline phase for the first student and the post-training phases for 
all students had just three data points (rather than five). In addition, the baseline sessions for 
each case only overlapped vertically for one probe data point. Finally, each case not receiv¬ 
ing the intervention did not have a probe point in a session in which another case either first 
received the intervention or reached the prespecified intervention criterion, but cases did 
continue to have baseline data after cases in higher tiers of the experiment entered the post¬ 
training phase. 11 For these three reasons, the experiments for both outcomes meet WWC pilot 
single-case design standards with reservations. 


Appendix A.2: Research details for De La Paz (1999) 

De La Paz, S. (1999). Self-regulated strategy instruction in regular education settings: Improving out¬ 
comes for students with and without learning disabilities. Learning Disabilities Research & Prac¬ 
tice, 14(2), 92-106. 


Setting 

The study was conducted in two middle schools in one school district in the southeastern 

Study sample 

United States. In one middle school, the student population was 94% White, 5% African 
American, and 1 % Asian or Latino; 18% of the students qualified for free or reduced-price 
lunch. In the other middle school, the student demographics were nearly identical, except only 
12% of its students qualified for free or reduced-price lunch. The intervention took place in 
two seventh-grade and one eighth-grade general education classrooms. 

The study sample included six students in seventh or eighth grade who were identified as 
having LD by their school district. The students had verbal IQ scores that ranged from 85 to 

125; scored below average on the reading, writing, or math sections of the Wechsler Individual 
Achievement Test (WIAT); 12 had no other disabilities; and spoke English as their primary lan¬ 
guage. One student was female, while the other five were male. All students in the study sample 
were White, and their ages ranged from 12 to 14 years old. The study also included separate 
experiments for 14 other students without LD, who were described as low-, average-, or high- 
achieving students. The experiments for students without LD are not described in this report or 
included in the ratings of effectiveness. 
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Intervention 

The SRSD intervention model was used to teach students how to write effective essays. 


The PLAN and WRITE mnemonics were used to help students remember the writing steps 
and strategies they had learned. The PLAN mnemonic asked students to pay attention to 
the writing prompt, list the main ideas, add supporting ideas, and number their ideas. The 
WRITE mnemonic asked them to work from the plan they had developed, remember their 
goals, include transition words, try to use different kinds of sentences, and use exciting words. 
Twelve to 16 sessions were offered, varying by classroom. Each SRSD session took place 
during one full class period for 4 days a week, over a 4-week period; some sessions were lost 
due to weather and other factors. Post-training essay probes were administered immediately 
following SRSD instruction. 

Comparison 

The study used a multiple probe design across classrooms for both outcomes. Each of the 
three classrooms had two students. During the baseline condition for each class, teachers 
taught their classes as usual. 

Outcomes and 
measurement 

The study outcomes were holistic quality and the number of words in each essay, which fall 
within the writing achievement domain. For a more detailed description of these outcome 
measures, see Appendix B. The study also measured the number of functional essay ele¬ 
ments; this outcome does not meet standards because data on inter-assessor agreement 
were not collected in each phase. 13 Results from the two experiments with outcomes in the 
writing achievement domain are presented in Appendix C.1. 

Support for 
implementation 

Teachers were given instruction manuals, and each attended 2 days of SRSD training. 

Author-reported 

findings 

The study authors found that the length of students’ essays (as measured by number of 
words) increased by 250%. They also found that the holistic quality generally improved, with 
the exception of one post-training essay for two students. The results of WWC’s correspond¬ 
ing visual analysis are presented in Appendix C. 

Maintenance 

Maintenance phases included one probe for each class and each outcome. In general, these 
probes were slightly lower than the mean of the post-training probes. 

WWC study rating 

Meets WWC Pilot Single-Case Design Standards With Reservations. 

Reason for study rating: The experiments for holistic writing and number of words used 
multiple probe designs; because the post-training phases had only three data points (rather 
than five) and some cases had data only in the first two baseline sessions (rather than the first 
three baseline sessions), these experiments meet WWC pilot single-case design standards 
with reservations. 
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Appendix A.3: Research details for De La Paz and Graham (1997) 

De La Paz, S., & Graham, S. (1997). Strategy instruction in planning: Effects on the writing performance 
and behavior of students with learning difficulties. Exceptional Children, 63, 167-181. 


Setting 

The study took place in two suburban mid-Atlantic elementary schools. The students received 


the intervention during individual instruction periods. 

Study sample 

The study sample included three fifth-grade students who received special education services 
for students with LD. Rand was an 11 -year-old African-American student who had an IQ of 64 
on the Wechsler Intelligence Scale-Ill. Elayne was an 11-year-old Caucasian student who had 
an IQ of 128. Aviendha was a 10-year-old African-American student who had an IQ of 105. 14 

Intervention 

The SRSD intervention model was used to teach students the STOP and DARE writing strate¬ 
gies. For STOP, students were taught to suspend judgment, take a side, organize ideas, and 
plan as they write. For DARE, students were taught to develop a topic sentence, add sup¬ 
porting ideas, reject possible arguments for the other side, and end with a conclusion. Each 
session lasted 45-55 minutes. Post-training essay probes were administered immediately 
following SRSD instruction. 

Comparison 

The study used a multiple probe design across participants. Prior to collecting baseline data, 
teachers had introduced students to the components of a good essay and the importance 
of advanced planning. During the baseline condition, students wrote essays on select topics 
without any planning instruction. 

Outcomes and 
measurement 

The study outcome was the number of functional essay elements, which falls within the writing 
achievement domain. For a more detailed description of this outcome measure, see Appendix 

B. The study also examined essay length, essay coherence, and holistic quality; these out¬ 
comes did not meet standards because they were not displayed graphically. In addition, the 
study authors measured the amount of time spent planning before writing, number of transfor¬ 
mations from planning to essay, writing time, strategy use, and social validation. These out¬ 
comes are not eligible under the study review protocol. Results from the one experiment with 
outcomes in the writing achievement domain are presented in Appendix C.1. 

Support for 
implementation 

The teacher used scripted lesson plans and a checklist to ensure that all instructional steps 
were completed. 

Author-reported 

findings 

The study authors found that the number of functional essay elements increased by 376% for 
Rand, 204% for Elayne, and 199% for Aviendha. The results of WWC’s corresponding visual 
analysis are presented in Appendix C. 

Maintenance 

All three students had one maintenance probe. Rand and Elayne’s probe was 6 weeks after 
the post-training phase, and Aviendha’s probe was 8 weeks after the post-training phase. In 
general, the values of these probes were in the same range as the post-training probes. 
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WWC study rating 

Meets WWC Pilot Single-Case Design Standards With Reservations. 

Reason for study rating: The study used a multiple probe design across participants to mea¬ 
sure the functional essay units outcome. The baseline sessions for each case only overlapped 
vertically for one probe data point (rather than three). In addition, each case not receiving the 
intervention did not have a probe point in a session in which another case either first receives 
the intervention or reaches the prespecified intervention criterion. 11 Thus, the experiment 
meets WWC pilot single-case design standards with reservations. 


Appendix A.4: Research details for Graham and Harris (1989) 

Graham S., & Harris K. R. (1989). Improving learning disabled students’ skills at composing essays: 
Self-instructional strategy training. Exceptional Children, 56(3), 201-214. 


Setting 

The study took place in one suburban elementary school in the northeastern United States. 


The intervention was administered individually to each student in a quiet room within the 
elementary school. 

Study sample 

The study sample included three sixth-grade students who were identified as having LD by 
their district and were receiving special education services in a resource room. Elaine was 
a 12-year-old girl with an IQ of 101 on the Wechsler Intelligence Scale for Children-Revised 
(WISC-R). She scored at least 3 years below grade level in math, writing, and knowledge 
achievement. Morgane was a 12-year-old girl with an IQ score of 89 on the Slossom Intel¬ 
ligence Test. She scored 2 years below grade level in reading. Arthur was a 12-year-old boy 
with an IQ of 99 on the WISC-R. He also scored 2 years below grade level in reading and math 
achievement and had repeated third grade. 

Intervention 

SRSD, referred to as self-instructional strategy training procedures in this study, was admin¬ 
istered individually to students to help improve their writing skills. The instructor, a graduate 
student majoring in special education, first worked with each student to define the compo¬ 
nents of a good essay using a mnemonic device (TREE) which prompted students to think 
of a topic sentence, reasons, examples, and an ending. The instructor then reviewed the 
student’s current level of essay writing and discussed goals. Next, the instructor presented 
a three-step essay-writing strategy to the student (Think, Plan, Write) and modeled for the 
student how to use the strategy. Next, the student memorized the strategy and practiced 
self-regulation. The instructor and the student then worked together to write an essay using 
the strategy. Finally, the student wrote essays independently. Elaine, Morgane, and Arthur 
participated in seven, five, and eight SRSD training sessions, respectively. Each session 
lasted approximately 40 minutes. Post-training essay probes were administered immediately 
following SRSD instruction. 

Comparison 

The study used a multiple probe design across three students. During the baseline condition 
for each student, the graduate student instructor asked students to write essays on a specific 
topic or in response to a picture. General procedures were in effect in the resource room dur¬ 
ing this period. 
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Outcomes and 
measurement 

The study outcome was the number of functional essay elements, which falls within the writing 
achievement domain. For a more detailed description of this outcome measure, see Appendix 

B. The study also examined four other eligible outcomes in the writing achievement domain 
that did not meet standards. These included the story grammar scale score, essay coherence, 
number of words written, and holistic quality. The story grammar scale outcome did not meet 
standards because there were fewer than three data points in baseline phases for Elaine and 
Morgane and in the post-training phase for Arthur. The other three outcomes did not meet 
standards because they were not presented graphically. Results from the one experiment with 
an outcome in the writing achievement domain are presented in Appendix C.1. 

Support for 
implementation 

The instructor received training on the step-by-step implementation procedures and received 
detailed lesson plans with guidance on how to conduct each step. 

Author-reported 

findings 

The study authors reported that the number of functional essay elements were greater for all 
three students after SRSD than they were at baseline. The results of WWC’s corresponding 
visual analysis are presented in Appendix C. 

Maintenance 

For Elaine, three maintenance writing probes were collected at 6, 11, and 12 weeks follow¬ 
ing the end of her training. Fler 6- and 12-week maintenance probes were similar in level to 
the post-training probes, but her 11-week maintenance probe was lower. For Morgane, one 
follow-up maintenance probe was collected 3 weeks following training. This probe was similar 
in level to the post-training probes. There were no maintenance probes for Arthur because the 
school year ended after his intervention was complete. 

WWC study rating 

Meets WWC Pilot Single-Case Design Standards With Reservations. 

Reason for study rating: The study used a multiple probe design across participants to mea¬ 
sure the number of functional essay elements. The baseline sessions for each case only over¬ 
lapped vertically for one or two data points (rather than three). In addition, some phases had 
only three or four data points (rather than five). Therefore, this experiment meets WWC pilot 
single-case design standards with reservations. 


Appendix A.5: Research details for Saddler (2006) 

Saddler, B. (2006). Increasing story-writing ability through self-regulated strategy development: Effects 
on young writers with learning disabilities. Learning Disability Quarterly, 29(4), 291-305. 


Setting 

The study was conducted in an urban elementary school in the northeastern United States. 

Study sample 

The school’s population was 50% White, 37% African-American, 7% Asian, and 6% Hispanic; 
48% of the students qualified for free or reduced-price lunch. The students’ intervention took 
place in a room outside of their general education classroom. 

There were six 7-year-old students in the sample from five second-grade classrooms. School 
staff identified the students as having LD while they were in first grade, and their current 
teachers identified them as struggling writers. The students were all African American; four 
were male (Mike, John, Larry, and George) and two were female (Tracy and Alice). All six 
students were reading at the first-grade level, and their full-scale IQ scores from the Stanford- 
Binet ranged from 89 to 102. The students received co-teaching support during their general 
education language arts classes from a special education teacher. 


Self-Regulated Strategy Development November 2017 


Page 23 




WWC Intervention Report 


Intervention 


Comparison 


Outcomes and 
measurement 


Support for 
implementation 

Author-reported 

findings 


The SRSD intervention used in this study included six lessons (some of which took multiple 
sessions) to help students improve their planning and story writing skills. A graduate student 
implemented the intervention to pairs of students. Students met with the instructor for 30 min¬ 
utes per session, three times a week. The training sessions took place outside of the student’s 
typical general education classroom, and students were still in their class during regular writ¬ 
ing instruction. The number of sessions was either 10 or 11, as each group moved through the 
lessons at their own pace. 

Lesson 1 focused on developing background knowledge, including introducing two mnemonic 
devices. POW reminded students to pick ideas, organize notes, and write and say more. The 
“WWW, What=2, How=2” device asked students to think about the following prompts: “Who 
are the main characters? When does the story take place? Where does the story take place? 
What do the main characters want to do? What happens when the main characters try to do 
it? How does the story end? How do the main characters feel?” Lesson 2 consisted of review 
and practice. Lesson 3 used self-statements designed to elicit story ideas. Lesson 4, which 
was repeated twice, involved a student-led collaborative writing exercise with the instructor. 
Lesson 5 involved another story writing exercise, but without the graphic organizer, and was 
repeated (two to three times) for each group until all seven story parts were included. Lesson 6 
involved a story writing exercise without instructor assistance. Post-training story probes were 
administered immediately following SRSD instruction. 

The study used multiple probe designs across pairs of students for each outcome. During the 
baseline condition, students wrote essays and participated in their regular writing instruction 
with their teachers in a general education classroom. 

The study outcomes were holistic quality, number of story elements, and number of words, 
which fall within the writing achievement domain. For a more detailed description of these 
outcome measures, see Appendix B. The study also measured planning time and quality of 
narratives. Planning time is not presented in the report because it does not fall under a domain 
specified in the protocol. The experiment measuring the effect on quality of narratives did not 
meet standards because data were not presented graphically. Results from the three experi¬ 
ments with outcomes in the writing achievement domain are presented in Appendix C.1. 

The study author trained the graduate student instructor. The author modeled each lesson and 
then observed the graduate student’s implementation of the lesson. 

The study author found that all students increased the number of story elements, but only one 
included all seven elements. They also found that all students increased their number of words 
and improved the holistic quality of their stories. The results of WWC’s corresponding visual 
analysis are presented in Appendix C. 
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Maintenance 

The author collected maintenance probes for all three pairs of students, 3 weeks after instruc¬ 
tion. The first two pairs also had a maintenance probe point 6 weeks after instruction. In gen¬ 
eral, the maintenance probes for the number of elements outcome were roughly similar with 
the post-training probe scores, although John scored consistently lower in maintenance than 
he did in post-training sessions. 

For the number of words outcome, Mike, Tracy, and Larry had similar performance during their 
post-training and maintenance phases. John and George scored lower during maintenance 
than post-training, with no overlap between the phases. Alice scored higher during mainte¬ 
nance than post-training, with no overlap. 

Finally, for the story quality outcome, all the students except George had similar performance 
during their post-training and maintenance phases. George scored lower on his one mainte¬ 
nance probe point, compared to the very consistent level in his three post-training probes. 

WWC study rating 

Meets WWC Pilot Single-Case Design Standards With Reservations. 

Reason for study rating: The experiments for all three outcomes used multiple probe designs 
across pairs of students; all three experiments receive the rating of meets WWC pilot single¬ 
case design standards with reservations because there are three data points (rather than five) in 
each baseline and post-training phase for all outcomes and students; there is just one baseline 
probe point within the first three sessions for Larry, Tracy, George, and Alice; and there are only 

2 baseline probes immediately before the intervention is introduced for Larry, Tracy, George, 
and Alice. In addition, each case not receiving the intervention did not have a probe point when 
the other cases entered the post-training period. Specifically, George and Alice (the third pair) 
did not have a probe point when Mike and John entered the post-training period. 11 


Appendix A.6: Research details for Saddler and Asaro (2007) 

Saddler, B., & Asaro, K. (2007). Increasing story quality through planning and revising: Effects on young 
writers with learning disabilities. Learning Disability Quarterly, 30(4), 223-234. 


Setting 

The study was conducted in an urban elementary school in the northeastern United States. 

Study sample 

The school’s student population was 52% White, 35% African-American, 7% Asian, and 6% 
Hispanic; approximately half of the students qualified for free or reduced-price lunch. The stu¬ 
dents’ intervention took place in a room outside of their general education classroom. 

There were six students in the sample, all of whom were 7 years old and in second grade. 

All students (three males, three females) were identified by their teacher as having LD in first 
grade, and their IQ scores ranged from 98 to 103. A special education co-teacher provided 
each student with additional support within their general education classroom. The sample 
included four African-American students and two White students. 
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Intervention 

The SRSD intervention model was used to teach students how to improve their story writing 


skills. The students participated in the SRSD intervention in pairs (Arnold and Maria; Gracie 
and George; and Scarlet and Rhett). The first and third pair covered the content of the inter¬ 
vention in 11 lessons, and the second pair needed 12 lessons. Lessons were taught three 
times a week in 30-minute sessions, outside of the general education classroom. 

The intervention included two strategies. The first was POW (pick ideas; organize notes; write 
and say more). During the “organize notes” stage of POW, the instructor also introduced the 
“WWW, What=2, How=2” mnemonic device which asked students to think about the follow¬ 
ing prompts: “Who are the main characters? When does the story take place? Where does 
the story take place? What do the main characters want to do? What happens when the main 
characters try to do it? How does the story end? How do the main characters feel?” Both 
strategies were introduced in the first lesson and were repeated and reviewed at the start of 
each subsequent lesson. The students began writing their own stories using the strategies 
beginning in Lesson 2, with growing independence over time as they became more comfort¬ 
able with the strategies. Post-training story probes were administered immediately following 
SRSD instruction. 

Comparison 

The study used a multiple baseline design across pairs of participants for each of the four eli¬ 
gible outcomes. During the baseline condition, students received their regular writing instruc¬ 
tion and wrote essays without receiving any strategy instruction. 

Outcomes and 
measurement 

The study examined four outcomes: holistic quality (of the first draft), number of revisions, num 
ber of story elements, and quality changes from first to revised draft, all of which fall within the 
writing achievement domain. For a more detailed description of these outcome measures, see 
Appendix B. The study also measured students’ planning time; this outcome is not eligible for 
review because it does not fall under a domain specified in the protocol. Results from the four 
experiments with outcomes in the writing achievement domain are presented in Appendix C.1. 

Support for 
implementation 

Not reported. 

Author-reported 

findings 

The study authors found that holistic quality, number of revisions, and number of story ele¬ 
ments all increased from baseline to post-training phases. They also reported that the quality 
of the revised draft was generally better than the first draft, with a few exceptions. The results 
of WWC’s corresponding visual analysis are presented in Appendix C. 

Maintenance 

There was no maintenance phase. 

WWC study rating 

Meets WWC Pilot Single-Case Design Standards With Reservations. 

Reason for study rating: The experiments for all four outcomes used multiple probe designs 
across pairs of participants. The experiments meet WWC pilot single-case standards with 
reservations because in each experiment, all six students’ post-training phases and Arnold, 
Maria, George, and Gracie’s baseline phases have three or four data points (rather than five). 

In addition, there is just one baseline probe point that overlaps vertically across all cases. 
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Appendix A.7: Research details for Sexton et al. (1998) 

Sexton, M., Harris, K. R., & Graham, S. (1998). Self-regulated strategy development and the writing pro 
cess: Effects on essay writing and attributions. Exceptional Children, 64(3), 295-311. 


Setting 

The study was conducted in a suburban mid-Atlantic school. The school’s student population 
was 62% African American, 23% White, 11 % Asian, and 3% Hispanic; about 40% of students 
were eligible for free or reduced-price lunch, and 8% were English learners. The students in 
this study received services in general education classes, with the support of special educa¬ 
tion teachers. 

Study sample 

The study sample included six students who were identified as having LD by their district. 
Marian, a 10-year-old fifth grader, was a Black female with an IQ of 81 and a third-grade read¬ 
ing level. Robin, a 10-year-old fifth grader, was a Black male with an IQ of 96 and a third-grade 
reading level. Alan, a 12-year-old sixth grader, was a White male with an IQ of 98 and a fourth- 
grade reading level. Matilda, an 11-year-old sixth grader, was a Black female with an IQ of 86 
and a third-grade reading level. Richard, a 10-year-old fifth grader, was a Black male with an 

IQ of 105 and a third-grade reading level. John, a 12-year-old sixth grader, was a Black male 
with an IQ of 117 and a fourth-grade reading level. 

Intervention 

The six-step SRSD intervention was used to help students write essays. Students were taught 
how to use the “TREE” mnemonic strategy which included starting with a topic sentence, 
stating the reasons behind their arguments, evaluating their reasons, and ending with a con¬ 
clusion. Once a stable baseline was obtained for both students in the first pair (Marian and 
Robin), SRSD instruction was started for that pair. Instruction for the second pair of students 
(Alan and Matilda) began once the first pair reached criterion level (one and a half times the 
number of functional elements produced during baseline). The same procedure was used for 
the third pair (Richard and John). The instruction period consisted of 40-50 minute sessions, 
and the number of sessions varied across pairs, ranging from 8-10 to complete the training. 

The first author of the study led the instruction. Post-training essay probes were administered 
immediately following SRSD instruction. 

Comparison 

The study used a multiple probe design across pairs of students. During the baseline period, 
usual writing instruction processes were used, in which students were encouraged to plan, 
draft, edit, and publish their papers. During the baseline condition, students wrote essays 
without special instruction. 

Outcomes and 
measurement 

The study examined the number of functional essay elements, which falls within the writing 
achievement domain. For a more detailed description of this outcome measure, see Appendix 

B. The study also measured essay length and holistic quality; these outcomes are not displayed 
graphically, so they do not meet WWC pilot single-case design standards. The study also mea¬ 
sured planning time, strategy use, and attributions; these outcomes are not eligible for review 
because they do not fall under a domain specified in the protocol. Results from the one experi¬ 
ment with an outcome in the writing achievement domain are presented in Appendix C.1. 

Support for 
implementation 

Not reported. 
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Author-reported 

findings 

The study authors reported that all six students increased their number of functional essay ele¬ 
ments from baseline to post-training, but that there was some overlap in scores for three of the 
students. The results of WWC’s corresponding visual analysis are presented in Appendix C. 

Maintenance 

The authors administered a maintenance probe for the first two pairs of students. The main¬ 
tenance probe points for Matilda and Robin were lower than their post-training probe points, 
while Marian and Alan’s maintenance probe points generally fell within the range of their post¬ 
training probe points. The third pair did not have any maintenance probes. 

WWC study rating 

Meets WWC Pilot Single-Case Design Standards With Reservations. 

Reason for study rating: The experiment for the number of functional essay elements used a 
multiple probe design across pairs of students. The experiment meets WWC pilot single-case 
design standards with reservations because there were at least three, but fewer than five, data 
points in several phases, and only one overlapping point across cases in the first three base¬ 
line sessions. In addition, Richard and John did not have a point in the first session after Alan 
and Matilda completed their SRSD instruction. 11 


Appendix A.8: Research details for Straub (2012) 

Straub, C. L. (2012). The effects of synchronous online cognitive strategy instruction in writing for 

students with learning disabilities (Doctoral dissertation). Retrieved from http://stars.library.ucf.edu/ 


Setting 

The study intervention took place in a secondary school in Florida, via an online tutoring 


system that connected the students to an instructor at a local university. The intervention 
occurred during a study hall period immediately after lunch. 

Study sample 

The study sample included four secondary school students who had been diagnosed with LD. 

All four students were White. Damon was a 14-year-old male in eighth grade; Carson was a 
15-year-old male in ninth grade; Jake was a 16-year-old male in tenth grade; and Monica was 
a 13-year-old female in sixth grade. 

Intervention 

SRSD was used to teach students strategies for both planning and writing. An undergraduate 
research assistant delivered the instruction via online chatting and video conferencing through 
the use of Adobe Connect and Google Docs. The structure of the intervention included up to 
five lessons, where the fifth lesson would be repeated if the student did not meet prespeci¬ 
fied criteria. Students were taught how to use the “POW+TREE” mnemonic strategies. POW 
reminded students to pick ideas, organize information, and then write and check their work. 
TREE prompted students to think of a topic sentence, reasons, examples, and an ending. 

During training, students could also earn points towards reinforcers. These reinforcers were 
described as “items valued at two dollars or less.” A preference assessment was used to 
develop a list of reinforcers, based on students’ preferences. Post-training essay probes were 
administered immediately following SRSD instruction. 

Comparison 

The study uses a multiple probe design across participants. During the baseline condition for 
each student, teachers taught their classes as usual. 
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Outcomes and 
measurement 

The study examined the number of correct minus incorrect word sequences and the number 
of functional essay elements, which both fall within the writing achievement domain. For a 
more detailed description of these outcome measures, see Appendix B. The study also mea¬ 
sured holistic quality and spontaneous writing; these outcomes are not displayed graphically, 
so they do not meet WWC pilot single-case design standards. There are also four ineligible 
outcomes. Social validity was measured but is not eligible for review because it does not 
fall under a domain specified in the protocol. Data were also collected on number of words 
written, number of seconds spent planning, and number of seconds spent writing, but these 
were not part of the primary analyses or used to measure impacts of SRSD, and thus are not 
eligible for review. Results from the two experiments with an outcome in the writing achieve¬ 
ment domain are presented in Appendix C.1. 

Support for 
implementation 

The tutor received 4 hours of training in applying the instructional procedures of SRSD. 

Author-reported 

findings 

The study author found that students increased both the number of correct minus incorrect 
word sequences and the number of functional essay elements. The author reported that all 
four students had marked changes in trend and level in both outcomes after receiving SRSD. 
The results of WWC’s corresponding visual analysis are presented in Appendix C. 

Maintenance 

Damon had two maintenance probes, Monica and Carson each had one, and Jake did not 
have any. Damon’s maintenance probe was consistent with the level of his tutoring phase 
probes, while the maintenance data for Monica and Carson show a drop-off in performance. 

WWC study rating 

Meets WWC Pilot Single-Case Design Standards With Reservations. 

Reason for study rating: The study uses a multiple probe design across participants for both 
outcomes. Carson’s tutoring phase had only four points, rather than five. In addition, each case 
had data points just prior to introduction of the independent variable, but only one of the cases 
(Damon) had three consecutive baseline probes. None of the cases had data points when prior 
cases either began the intervention or reached a prespecified level. 11 Thus, the experiments for 
both outcomes meet WWC pilot single-case design standards with reservations. 


Appendix A.9: Research details for Troia et al. (1999) 

Troia, G. A., Graham, S., & Harris, K. R. (1999). Teaching students with learning disabilities to mindfully plan 
when writing. Exceptional Children, 65(2), 235-252. Retrieved from https://eric.ed.gov/7idsEJ579491 


Setting 

The study took place in two suburban elementary schools located in the mid-Atlantic United 

Study sample 

States. 

The study sample included three students (Luke, Ben, and Leia) from two schools. All three 
students were in the fifth grade, had an IQ of 101, were from an economically disadvantaged 
background, spoke English as their primary language, and were identified by their school dis¬ 
trict as having LD. 

Luke was a 10-year-old African-American male. Ben was an 11-year-old White male who 
had been retained in second grade. Leia was an 11-year-old White female who had repeated 
first grade. 
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Intervention 

SRSD was used to teach students how to integrate planning strategies into the writing pro¬ 
cess. Using the SRSD intervention, the instructor modeled how to use three strategies in the 
first three lessons: goal setting, brainstorming, and organizing. In the first lesson, the instruc¬ 
tor prepared a speech and wrote a story while modeling aloud how to use the strategies. The 
instructor gave students a list of questions to help them identify essential features and the 
value of the strategies used by the instructor. The second lesson followed the same proce¬ 
dures, but the instructor instead read a chapter and wrote a story, and the third lesson fol¬ 
lowed the same procedures as the first two lessons, with the instructor planning a trip. In the 
second and third lessons, the students were asked to compare the strategy usage with earlier 
lessons. The instructor also introduced the STOP & LIST (Stop, Think of Purpose & List Ideas, 
Sequence Them) strategies to the students. The fourth and fifth lessons focused on applying 
the STOP & LIST strategies. In lessons 6 and 7, the students were expected to plan and write 
stories independently. The seven lessons took 3 weeks to cover, in 60-90 minute sessions. 
Post-training story probes were administered immediately following SRSD instruction. 

Comparison 

The study used a multiple probe design across participants. During the baseline condition for 
each student, teachers taught their classes as usual. 

Outcomes and 
measurement 

The study examined the story grammar scale score, which falls within the writing achievement 
domain. For a more detailed description of this outcome measure, see Appendix B. The study 
also measured the number of functional essay elements; this outcome did not meet WWC stan¬ 
dards because there are insufficient data to evaluate the attempts to demonstrate an intervention 
effect. In addition, the study measured eight other outcomes that do not meet WWC standards 
because they are not displayed graphically: number of propositions in planning stories; number 
of propositions in planning essays; strategy use when writing; strategy use in final drafts; length 
of story; length of essay; essay holistic quality; and story holistic quality. The study also reported 
five ineligible outcomes: planning time for stories, writing time for stories, planning time for 
essays, writing time for essays, and social validation. These outcomes are not eligible for review 
because they do not fall under a domain specified in the protocol. Results from the one experi¬ 
ment with an outcome in the writing achievement domain are presented in Appendix C.1. 

Support for 
implementation 

Not reported. 

Author-reported 

findings 

The study authors found that the students increased their story grammar scale scores from 
baseline to post-training. The results of WWC’s corresponding visual analysis are presented in 
Appendix C. 

Maintenance 

Each student had one maintenance probe 3 weeks after their last post-training probe point. 
Ben’s maintenance probe was at a comparable level to his post-training probes, but Luke and 
Leia scored lower in maintenance than they did in the post-training phase. 

WWC study rating 

Meets WWC Pilot Single-Case Design Standards With Reservations. 

Reason for study rating: The experiment used a multiple probe design across participants, 
and there were fewer than five data points in at least one phase; because all phases had at 
least three data points, but less than five, this experiment meets WWC pilot single-case design 
standards with reservations. In addition, none of the cases had data points when prior cases 
either began the intervention or reached a prespecified level. 11 
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Appendix B: Outcome measures for the writing achievement domain 


Writing achievement 


Writing achievement 


Holistic quality 

Holistic quality was measured in multiple studies, but the measurement and definition of the outcome varied: 

In De La Paz (1999), two general education teachers reviewed and scored all of the essays, on a scale of 0 
(low quality) to 7 (high quality), which represents a general impression of overall quality. The teachers were 
given representative examples of low, average, and high quality essays, and were asked to consider the ideas 
portrayed, the development, organization, unity, and coherence of the essay, and the quality of the vocabulary 
used. Essays were typed and corrected for spelling, punctuation, and capitalization prior to review. Any differ¬ 
ences between reviewers were discussed, and the resulting score was based on mutual agreement (as cited in 

De La Paz, 1999). 

In Saddler (2006), two graduate students reviewed and scored all stories on a scale of 1 (low quality) to 8 
(high quality), which represents a general impression of overall story quality. Scorers were given representative 
examples of scores 2, 4, and 6. Essays were typed prior to review, and no changes were made to spelling, 
punctuation, or capitalization. The average score across the two reviewers was used as the measure of holistic 
quality (as cited in Saddler, 2006). 

In Saddler and Asaro (2007), two examiners unfamiliar with the study scored the first draft of all compositions 
on a scale of 1 (low quality) to 8 (high quality), based on the measure developed by Graham and Harris (1989) 15 
to assess overall story quality. Scorers were given representative examples of scores 2, 4, and 7. Stories were 
typed prior to review, and no corrections were made to spelling, punctuation, or capitalization. The average 
score across the two reviewers was used as the measure of holistic quality (as cited in Saddler & Asaro, 2007). 

Number of correct minus incorrect word 
sequences 

This measure was used to assess improvements in writing, and captured changes in frequency of spelling, 
punctuation, and grammatical errors. Higher values represent a higher degree of improvement (as cited in 

Straub, 2012), 

Number of functional essay elements 

Number of functional essay elements was measured in multiple studies, but the measurement and definition of 
the outcome varied: 

In three of the studies, the total number of functional essay elements outcome assessed the use of four ele¬ 
ments recommended by Graham (1990) 16 and Scardamalia, Bereiter, and Goelman (1982) 17 : premise, reason, 
conclusion, and elaboration. Essay elements were deemed functional if they directly supported the development 
of the student’s argument. For example, in order for a statement to be scored as a functional "premise,” the 
statement had to fully explain the position the student was trying to make in the essay. "Reasons” needed to 
explain why the writer believed a particular premise. "Conclusions” were closing statements that brought every¬ 
thing together, and “elaborations” included text that supported or exemplified premises, reasons, or conclusions. 
These functional text units were summed to form the outcome. Nonfunctional text referred to text that was 
repeated without but had no purpose (as cited in De La Paz & Graham, 1997 a ; Graham & Harris, 1989 a ; and 
Sexton, Harris, & Graham, 1998). 

In Straub (2012), researchers followed the scoring format used by Mason, Kubina, and Taft (2009). 18 The 
number of functional essay elements assessed the inclusion of four elements: belief/topic sentence, reason, 
explanation, and ending statement. One point was awarded for each element that was included, and extra points 
were awarded for each additional reason or explanation, resulting in a potential score of eight points or more (as 
cited in Straub, 2012). 

Number of revisions 

The number of revisions made to stories included changes in spelling, punctuation, capitalization, as well as the 
addition, deletion, or substitution of text. Each word that was changed, added, or deleted counted as a single 
revision (as cited in Saddler & Asaro, 2007). 

Number of story elements 

Number of story elements was measured similarly in multiple studies. Two examiners unfamiliar with the study 
scored the number of story elements on a 0-7 scale. This score, which was used as a measure of story com¬ 
pleteness, was designed by Graham and Harris (1989). The reviewers tabulated the inclusion of seven common 
elements that students included in their paper, including the main character, locale, time, goal, action, ending, 
and reaction (as cited in Danoff, Harris, & Graham, 1993; Saddler, 2006; and Saddler & Asaro, 2007). 

Number of words 

Number of words was measured in multiple studies, but the measurement and definition of the outcome varied: 

The total number of words was captured by word processing software after essays were typed. All words that 
represented a spoken word were counted regardless of any spelling errors (as cited in De La Paz, 1999). 

The total number of words was tabulated after essays were typed. All words, except for those in the title, were 
counted regardless of any spelling errors (as cited in Saddler, 2006). 
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Quality of changes from first to revised Two graduate students reviewed stories and scored changes in quality between drafts, following procedures 
draft from MacArthur and Graham (1987). 19 This score is an indicator of the effectiveness of the revisions made by 

the students. Raters used the first draft as the standard and rated the second draft in comparison, using a scale 
of -2 (the second draft was much worse than the first) to +2 (the second draft was much better than the first) 
(as cited in Saddler and Asaro, 2007). 

Story grammar scale score The story grammar scale score, based on a measure designed by Graham and Harris (1989), 20 was measured 

in multiple studies, but the measurement and definition of the outcome varied: 

In Danoff, Harris, and Graham (1993), the scale was used to assess the quality of eight story elements: main 
character, locale, time, starter event, goal, action, ending, and reaction. For each story element, a score of 0 
was awarded if the element was not present in the student's story, a score of 1 was awarded if the element was 
present, and a score of 2 was awarded if the element was highly developed. Additional points were awarded 
if two or more goals were articulated or if stories contained well-defined episodes or included events that 
happened in a logical order. Scores for the individual story elements were totaled to obtain a total story grammar 
scale score, with a maximum of 19 points (as cited in Danoff, Harris, & Graham, 1993). 

In Troia, Graham, and Harris (1999), this scale was used to evaluate the schematic structure of the stories, 
including the inclusion and quality of 10 elements: main character, locale, time, initiating event, goal, attempt to 
achieve goal, consequence of attempt, reaction, dialogue, and title. For each story element other than dialogue, 
a score of 0 was awarded if the element was not present in the student’s story, a score of 1 was awarded if 
the element was present, and a score of 2 was awarded if the element was highly developed. For dialogue, 
the score ranged from 0 (dialogue not present) to 1 (dialogue present). For all elements other than dialogue, 
additional points were awarded if the story contained more than one episode (as cited in Troia, Graham, & 

Harris, 1999). a 

a The authors collected inter-assessor agreement (IAA) data in each phase and on at least 20% of all sessions, but it is not clear if IAA data were collected during 20% of the data 
points in each condition. 
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Appendix C: Single-case design findings included in the effectiveness ratings 


Table C: Single-case design findings for the writing achievement domain 



Study characteristics 



WWC summary 







Intervention effects 

Outcome measure 

Sample 
size (case) 

Age(s) 

Design type 

Evidence 

level 

Total 

demonstrated 

Total 

attempted 

Danoff, Harris, & Graham (1993) a 

Number of story elements 

3 (Students with learning 
disabilities subgroup) 

Grades 4 
and 5 

Multiple probes 
across participants 

Strong (+) 

3 

3 

Story grammar scale score 

3 (Students with learning 
disabilities subgroup) 

Grades 4 
and 5 

Multiple probes 
across participants 

Strong (+) 

3 

3 

De La Paz (1999) b 

Holistic quality 

3 (Classes taught by Graham, 
Parker, and Owen, with two 
students in each class) 

12-14 

Multiple probes 
across participants 

No 

evidence 

2 

3 

Number of words 

3 (Classes taught by Graham, 
Parker, and Owen, with two 
students in each class) 

12-14 

Multiple probes 
across participants 

Strong (+) 

3 

3 

De La Paz & Graham (1997) 

Number of functional essay 
elements 

3 (Rand, Elayne, and 
Aviendha) 

10-11 

Multiple probes 
across participants 

Strong (+) 

3 

3 

Graham & Harris (1989) 

Number of functional essay 
elements 

3 (Elaine, Morgane, Arthur) 

12 

Multiple probes 
across participants 

Strong (+) 

3 

3 

Saddler (2006) c 

Holistic quality 

6 (Mike, John, Tracy, Larry, 
George, and Alice) 

7 

Multiple probes 
across participants 

Strong (+) 

6 

6 

Number of story elements 

6 (Mike, John, Tracy, Larry, 
George, and Alice) 

7 

Multiple probes 
across participants 

Strong (+) 

6 

6 

Number of words 

6 (Mike, John, Tracy, Larry, 
George, and Alice) 

7 

Multiple probes 
across participants 

Strong (+) 

6 

6 

Saddler & Asaro (2007) d 

Holistic quality (of first draft) 

6 (Arnold, Maria, Gracie, 
George, Scarlett, and Rhett) 

7 

Multiple probes 
across participants 

Strong (+) 

6 

6 

Number of revisions 

6 (Arnold, Maria, Gracie, 
George, Scarlett, and Rhett) 

7 

Multiple probes 
across participants 

Strong (+) 

6 

6 

Number of story elements 

6 (Arnold, Maria, Gracie, 
George, Scarlett, and Rhett) 

7 

Multiple probes 
across participants 

Strong (+) 

6 

6 

Quality of changes from first to 
revised draft 

6 (Arnold, Maria, Gracie, 
George, Scarlett, and Rhett) 

7 

Multiple probes 
across participants 

No 

evidence 

3 

6 

Sexton, Harris, & Graham (1998) e 

Number of functional essay 
elements 

6 (Marian, Robin, Alan, 
Matilda, Richard, and John) 

10-12 

Multiple probes 
across participants 

Moderate 

(+) 

3 

6 
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Study characteristics WWC summary 

Intervention effects 

Sample Evidence Total Total 


Outcome measure 

size (case) 

Age(s) 

Design type 

level 

demonstrated 

attempted 

Straub (2012) 

Number of correct minus incorrect 
word sequences 

4 (Damon, Carson, Jake, and 
Monica) 

13-16 

Multiple probes 
across participants 

Strong (+) 

4 

4 

Number of functional essay 
elements 

4 (Damon, Carson, Jake, and 
Monica) 

13-16 

Multiple probes 
across participants 

Strong (+) 

4 

4 

Troia, Graham, & Harris (1999) 

Story grammar scale score 

3 (Luke, Ben, and Leia) 

10-11 

Multiple probes 

Strong (+) 

3 

3 


across participants 


Table Notes: The WWC does not calculate effect sizes for single-case design research. Characterizations of Strong and Moderate evidence, based on WWC visual analysis, indi¬ 
cate that the experiment demonstrated an effect of the intervention. Characterizations of No evidence indicate that the experiment did not provide at least three demonstrations of 
an intervention effect in the same direction. + = a positive (favorable) effect in the desired direction. 
a In Danoff et al. (1993), the ages of students were not provided. 

b In De La Paz (1999), each experiment included three classrooms (or cases), with two students within each classroom, for a total of six students across three total cases. In the 
experiments for both outcomes, each probe point reflects the average score of the two students in a classroom. 

c In Saddler (2006), each experiment involved introducing the intervention to three pairs of students. Mike and John received SRSD first, then Tracy and Larry, and then George and 
Alice. Thus, there are six attempts to demonstrate an effect at three different points in time. 

d In Saddler and Asaro (2007), each experiment involved introducing the intervention to three pairs of students. Arnold and Maria received SRSD first, then George and Grade, and 
then Scarlett and Rhett. Thus, there are six attempts to demonstrate an effect at three different points in time. The experiment for the outcome measuring quality changes from first 
to revised drafts was characterized as providing No evidence following WWC visual analysis. The WWC Procedures and Standards Handbook (version 3.0) characterizes experiments 
as providing Moderate evidence when the experiment has at least three demonstrations of an effect at three different points in time, and at least one demonstration of a non-effect. 
This experiment had at least three demonstrations of an effect, but the effects occurred at only two points in time, so the experiment was characterized as providing No evidence. 
For the number of story elements outcome, an author query confirmed that Maria had three baseline probe points, even though one was missing from the graph in the original 
study; she scored a 2 on this missing probe point. The authors also confirmed that all students had at least three data points in each phase for each of the outcomes that meet 
review requirements. 

e In Sexton et al. (1998), the experiment involved introducing the intervention to three pairs of students. Marian and Robin received SRSD first, then Alan and Matilda, and then Richard 
and John. Thus, there are six attempts to demonstrate an effect at three different points in time. 


Self-Regulated Strategy Development November 2017 


Page 34 







WWC Intervention Report 


Appendix D: Single-case design findings in a domain not included in the effectiveness ratings 21 

Table D.l. Research details for single-case design studies with outcomes in the math achievement domain 


Study Study sample, setting, comparison sessions, and intervention sessions 


Case, Harris, & Graham This study included four children (Ben, Abernathy, Willow, and Paladin) with learning disabilities. All four students were 11 
(1992), Meets WWC Pilot years old and in self-contained fifth- or sixth-grade classrooms in an urban elementary school in the northeastern United 

Single-Case Design States. Students had IQ scores between 77 and 82 on the Wechsler Intelligence Scale for Children-Revised (WISC-R) and 

Standards With Reservations achievement at least 2 years below grade level on one or more academic areas. Paladin’s contrast was not included in the 

current review, as his baseline phase was not concurrent with the baseline phases of the other three students. 

The study used a multiple baseline design experiment across participants. The baseline condition consisted of normal 
classroom practice. There were two successive SRSD intervention conditions: addition instruction and subtraction instruc¬ 
tion, both of which used SRSD procedures that focused on self-assessment and self-recording. The addition instruction 
was given first, and the subtraction instruction was given after addition outcomes were measured. Due to possible 
carryover or residual treatment effects from the addition intervention, the subtraction outcomes measured after the sub¬ 
traction instruction cannot meet WWC pilot single-case design standards because the measures of effectiveness cannot 
be attributed solely to the subtraction intervention. Thus, the current review focuses on the addition SRSD instruction and 
addition outcomes. 

SRSD was used to improve students' mathematical problem-solving skills. Instruction included asking students to list 
words that indicated when addition (or subtraction) should be used; the instructor gave students cards with vocabulary 
words and examples to help students learn key phrases and cue words within word problems. The instructor had a confer¬ 
ence with each student to discuss their performance and the goals of the instruction and then introduced the strategy (for 
example, reading the problem out loud, looking for and circling important words, drawing a picture, writing a sentence, 
and writing the answer). The instructor modeled the strategy using a think-aloud process, worked through problems with 
the student, and then asked the student to independently use the strategy to solve problems. The addition intervention 
was implemented during 35-minute sessions, two-to-three times a week. Ben, Abernathy, and Willow received 165,130, 
and 180 minutes of addition instruction, respectively. 

For all three students, addition outcomes included number of correct equations and number of correct equations and 
answers, both of which fall under the math achievement domain. For a more detailed description of these outcome 
measures, see Appendix D, Table D.2. 

In the multiple baseline design across participants, there were fewer than five data points in at least one phase, so this 
study meets WWC pilot single-case design standards with reservations. 


Table D.2. Outcome measures in single-case design studies for the math achievement domain 


Math achievement 

Number of correct equations 

The number of correct equations measured the degree to which students understood word problems. The score 
represents the number of times a student provided a correct addition equation, based on a word problem. This 
outcome ranged from 0 to 7, with higher values representing higher degrees of understanding (as cited in Case, Harris, 

& Graham, 1992). a 

Number of correct equations 
and answers 

This outcome measured the degree to which students understood word problems and correctly solved equations. The 
score represents the number of times a student provided a correct addition equation, based on a word problem, and 
then correctly solved the equation. This outcome ranged from 0 to 7, with higher values representing higher degrees of 
understanding and correct answers (as cited in Case, Harris, & Graham, 1992). a 


a The authors collected inter-assessor agreement (IAA) data in each phase and on at least 20% of all sessions, but it is not clear whether IAA data were collected during 20% of the 
data points in each condition. 
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Table D.3: Single-case design findings for the math achievement domain 



Study characteristics 

WWC summary 

Outcome measure 

Sample size (case) 

Age(s) 

Design type 

Evidence level 

Case, Harris, & Graham (1992) a 

Number of correct equations 

3 (Ben, Abernathy, Willow) 

11 

Multiple probes across 
participants 

No evidence 

Number of correct equations and 
answers 

3 (Ben, Abernathy, Willow) 

11 

Multiple probes across 
participants 

No evidence 


Table Notes: The WWC does not calculate effect sizes for single-case design research. Characterizations of Strong and Moderate evidence, based on WWC visual analysis indicate 
that the experiment demonstrated an effect of the intervention. Characterizations of No evidence indicate that the experiment did not provide at least three demonstrations of an 
intervention effect in the same direction. The evidence from the SCD studies on SRSD does not reach the threshold to include SCD evidence in the effectiveness ratings for the 
math achievement domain. 

a For Case et al. (1992), the study used a multiple baseline design experiment across participants to measure the effect of two successive SRSD intervention conditions: addition 
instruction and subtraction instruction. The addition instruction was given first, and the subtraction instruction was given after addition outcomes were measured. Due to a possible 
carryover or residual treatment effect from the addition intervention, the subtraction outcomes measured after the subtraction instruction cannot meet WWC pilot single-case design 
standards because the measures of effectiveness cannot be attributed solely to the subtraction intervention. Therefore, the subtraction outcomes are not included in this report. In 
addition, Paladin’s contrast was not included in the current review, as his baseline phase was not concurrent with the baseline phases of the other three students. 
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Endnotes 

1 The descriptive information for this intervention was obtained from Harris et al. (2003) and Santangelo et al. (2008). The What Works 
Clearinghouse (WWC) requests developers review the intervention description sections for accuracy from their perspective. The WWC 
provided the developers with the intervention description in January 2015, and the WWC incorporated feedback from the developers. 
Further verification of the accuracy of the descriptive information for this intervention is beyond the scope of this review. Full citations: 
Harris, K. R., Graham, S., & Mason, L. H. (2003). Self-regulated strategy development in the classroom: Part of a balanced approach 
to writing instruction for students with disabilities. Focus on Exceptional Children, 35(7), 1-16; Santangelo, T., Harris, K. R., & Graham, 
S. (2008). Using self-regulated strategy development to support students who have ‘trubol giting thangs into werds.’ Remedial and 
Special Education, 29(2), 78-89. 

2 The literature search reflects documents publicly available by February 2017. Studies in this report were reviewed using the WWC 
Procedures and Standards Handbook (version 3.0) and the Students With a Specific Learning Disability review protocol (version 
3.0). One study, Curry (1997), was reviewed previously for the WWC practice guide on Teaching Elementary Students to Be Effective 
Writers (2012), and met WWC group design standards with reservations. The review for the current report resulted in a revised rating 
of Does Not Meet WWC Group Design Standards because baseline differences between the analytic intervention and comparison 
groups are in the range where the WWC requires a statistical adjustment for the baseline measure, but an acceptable statistical 
adjustment was not performed. The author adjusted for baseline differences using an ANCOVA but did not report adjusted means 
or describe the direction of the impact. The WWC Reviewer Guidance for the WWC Procedures and Standards Handbook (version 
3.0) clarifies that when the WWC requires a statistical adjustment to be performed, but a study does not describe the direction of the 
finding after the adjustment from a credible analysis, the finding is rated Does Not Meet WWC Group Design Standards. The author 
did not respond to a request for information about the direction, magnitude, and statistical significance of the adjusted finding. The 
prior review for the WWC practice guide was conducted using the WWC Procedures and Standards Handbook (version 2.0), which 
specified that a difference-in-differences adjustment applied by the WWC was an acceptable approach for satisfying the statisti¬ 
cal adjustment requirement, but this approach is not acceptable under the WWC Procedures and Standards Handbook (version 
3.0). The evidence presented in this report is based on available research. Findings and conclusions could change as new research 
becomes available. 

3 For the math achievement domain, there is one study (fewer than the five required), one research team (fewer than the three 
required), and three cases (fewer than the 20 required). 

4 Please see the Students With a Specific Learning Disability review protocol (version 3.0) for a list of all outcome domains. 

5 For criteria used to determine the rating of effectiveness for single-case design studies, see the WWC Rating Criteria on p. 39. 

6 The results from single-case design studies are not used to report an intervention effectiveness rating for an outcome domain unless 
the studies collectively meet the threshold criteria described on p. 39. The evidence from the single-case design studies on SRSD 
does not reach the threshold to include single-case design evidence in the effectiveness ratings for the math achievement domain. 

7 In single-case design research, a case, such as a student or classroom, is the unit of intervention administration and data analysis. 

A single-case design experiment examines a single outcome measure repeatedly within and across different phases defined by the 
presence or absence of the intervention. There might be multiple experiments for a case if more than one outcome is examined, for 
example. All experiments within a research article are considered as one single-case design study. 

8 The results from single-case design studies are not used to report an intervention effectiveness rating for an outcome domain unless 
the studies collectively meet the threshold criteria described on p. 39. The evidence from the single-case design studies on SRSD 
does not reach the threshold to include single-case design evidence in the effectiveness ratings for the math achievement domain. 

9 In single-case design research, a case, such as a student or classroom, is the unit of intervention administration and data analysis. 

A single-case design experiment examines a single outcome measure repeatedly within and across different phases defined by the 
presence or absence of the intervention. There might be multiple experiments for a case if more than one outcome is examined, for 
example. All experiments within a research article are considered as one single-case design study. 

10 When there is more than one single-case design experiment in a publication that does not meet WWC pilot single-case design 
standards, the citation list reports the disposition code that applies to the majority of single-case designs in that publication. Some 
single-case design experiments within a given publication might not meet WWC pilot single-case design standards for reasons other 
than the one listed in the citation list. 
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11 The WWC Procedures and Standards Handbook includes three criteria that multiple probe designs must meet in order to Meet 
WWC Pilot Single-Case Design Standards With or Without Reservations. These additional criteria are required because some baseline 
data points are intentionally missing in multiple probe designs. One of these criteria requires that each case not receiving the interven¬ 
tion must have a probe point in a session where another case either (a) first receives the intervention or (b) reaches the prespecified 
intervention criterion. However, as specified in the Students With a Specific Learning Disability topic area protocol, studies can Meet 
WWC Pilot Single-Case Design Standards With Reservations, even if they do not meet this requirement. Cases must still continue to 
have baseline data for at least one session after the intervention is administered to preceding cases, as this is a requirement for all 
multiple baseline designs, and must meet the other two multiple probe criteria specified in the Handbook. 

12 Five of the students scored at least one standard deviation below average on the WIAT, and the remaining student scored in the low 
average range of the WIAT, rather than a full standard deviation below average. 

13 For De La Paz (1999), a response to an author query confirmed that data on inter-assessor agreement were not collected in each phase. 

14 Single-case design studies typically assign participants a pseudonym; we use the pseudonyms provided by study authors in this 
report so that WWC ratings can be easily mapped to the correct single-case design in the original study. 

15 Graham, S., & Harris, K. R. (1989). A components analysis of cognitive strategy training: Effects on learning disabled students’ com¬ 
positions and self-efficacy. Journal of Educational Psychology, 87, 353-361. 

16 Graham, S. (1990). The role of production factors in learning disabled students’ compositions. Journal of Educational Psychology, 

82, 781-791. 

17 Scardamalia, M., Bereiter, C., & Goelman, H. (1982). The role of production factors in writing ability. In M. Nystrand (Ed.), What writ¬ 
ers know: The language, process, and structure of written discourse. New York, NY: Academic Press. 

18 Mason, L. H., Kubina, R. M., & Taft, R. J. (2009). Developing quick writing skills of middle school students with disabilities. Journal of 
Special Education, 44(4), 205-220. 

19 MacArthur, C. A., & Graham, S. (1987). Learning disabled students’ composing under three methods of text production: Handwrit¬ 
ing, word processing, and dictation. Journal of Special Education, 21(3), 22-42. 

20 Graham, S., & Harris, K. R. (1989). A components analysis of cognitive strategy training: Effects on learning disabled students’ com¬ 
positions and self-efficacy. Journal of Educational Psychology, 87, 353-361. 

21 The results from single-case design studies are not used to report an intervention effectiveness rating for an outcome domain unless 
the studies collectively meet the threshold criteria described on p. 39. The evidence from the single-case design studies on SRSD 
does not reach the threshold to include single-case design evidence in the effectiveness ratings for the math achievement domain. 
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WWC Rating Criteria 


Criteria used to determine the rating of a study that includes single-case experiments 


Study rating Criteria 


Meets WWC pilot single- A single-case design study that provides the highest degree of confidence for assessing an intervention’s effectiveness. 

case design standards 
without reservations 

Meets WWC pilot single- A study that provides a lower degree of confidence for assessing an intervention's effectiveness, such as a reversal- 

case design standards withdrawal design with three or four data points per phase, rather than five. 

with reservations 


Criteria used to determine evidence of a causal relation in a single-case design experiment 


Evidence level Criteria 


Strong evidence of a A single-case design study with at least three demonstrations of the intervention effect and no non-effects. 

causal relationship 

Moderate evidence of a A single-case design study with at least three demonstrations of the intervention effect and at least one non-effect. 

causal relationship 

No evidence of a causal A single-case design study with fewer than three demonstrations of the intervention effect. 

relationship 


Criteria used to determine whether there is enough research for single-case design evidence to contribute 
to a WWC effectiveness rating for a given domain 


Threshold to include 

single-case design evidence Criteria 


Threshold met At least five studies examining the intervention meet WWC pilot single-case design standards without reservations or 

meet WWC pilot single-case design standards with reservations, AND 

The single-case design studies are conducted by at least three different research teams with no overlapping author¬ 
ship at three different institutions, AND 

The combined number of cases (i.e., participants, classrooms, etc.) totals at least 20. 


Criteria used to determine the rating of effectiveness for an intervention based on single-case design research 


Rating of effectiveness Criteria 


Positive effects Across all single-case design experiments, at least 80% show positive effects, AND 

No single-case design experiment shows negative effects, AND 

At least one single-case design experiment meets WWC pilot single-case design standards without reservations. 


Potentially positive effects Across all the single-case design experiments, 51% to 79% show positive effects, AND 

No single-case design experiment shows negative effects. 


Mixed effects 


Potentially negative effects 


At least one single-case design experiment shows positive effects AND at least one single-case design experi¬ 
ment shows negative effects, OR 

At least one single-case design experiment shows positive or negative effects AND 50% or more show indeter¬ 
minate effects. 

Across all the single-case design experiments, 51% to 79% show negative effects, AND 
No single-case design experiment shows positive effects. 
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WWC Rating Criteria 

1 Rating of effectiveness 

Criteria 

Negative effects 

Across all the single-case design experiments, at least 80% show negative effects, AND 

No single-case design experiment shows positive effects, AND 


At least one single case design experiment meets WWC pilot single-case design standards without reservations. 

No discernible effects None of the single-case design experiments shows effects, either positive or negative. 

Notes: A single-case design experiment has all of the design elements required to meet WWC standards with or without reservations (such as three attempts to demonstrate an 
effect) and is presented as one experiment in a study. The WWC characterizes all single-case design experiments in the same research article as one study, and thus one study 
can have multiple single-case design experiments. For example, a study could include three separate ABAB design experiments for one student (across three different eligible 
outcomes) or could include three separate ABAB design experiments for three separate eligible students. If a study presents data for more than one outcome, the WWC clas¬ 
sifies the single-case design for each outcome as a separate experiment. The WWC visual analysis characterizations of Strong and Moderate evidence indicate that the design 
demonstrated an effect of the intervention. A visual analysis rating of No evidence indicates that the experiment did not provide at least three demonstrations of an intervention 
effect in the same direction. 
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Glossary of Terms 

ABAB design 

Alternating treatment 
design 

Attrition 


Baseline 

Case 

Confounding factor 

Demonstration of an 
intervention effect 

Design 


Effect size 


This is an example of a reversal-withdrawal single-case design in which there are four phases: 
a baseline (A) followed by an introduction of the intervention (B), a withdrawal of the interven¬ 
tion to return to the baseline condition (A), and a second introduction of the intervention (B). 

A single-case design experiment that rapidly alternates between two or more interventions to 
examine how outcomes change; outcomes are measured with only one or two data points 
within each phase. 

Attrition occurs when an outcome variable is not available for all subjects initially assigned 
to the intervention and comparison groups. If a randomized controlled trial (RCT) or regres¬ 
sion discontinuity design (RDD) study has high levels of attrition, the validity of the study 
results can be called into question. An RCT with high attrition cannot receive the highest 
rating of Meets WWC Group Design Standards Without Reservations, but can receive a 
rating of Meets WWC Group Design Standards With Reservations if it establishes baseline 
equivalence of the analytic sample. Similarly, the highest rating an RDD with high attrition 
can receive is Meets WWC RDD Standards With Reservations. 

For single-case design research, attrition occurs when an individual fails to complete all 
required phases or data points in an experiment, or when the case is a group and indi¬ 
viduals leave the group. If a single-case design does not meet minimum requirements for 
phases and data points within phases, the study cannot receive the highest rating of Meets 
WWC Pilot Single-Case Design Standards Without Reservations. 

A point in time before the intervention was implemented in group design research and in regres¬ 
sion discontinuity design studies. When a study is required to satisfy the baseline equivalence 
requirement, it must be done with characteristics of the analytic sample at baseline. In a single¬ 
case design experiment, the baseline condition is a period during which participants are not 
receiving the intervention. 

A case is the unit of intervention administration and data analysis in a single-case design 
experiment. A case may be a single participant or a group of participants, such as a classroom. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

In single-case design research, an effect is demonstrated when the data pattern in one 
phase (e.g., an intervention phase) differs more than would be expected from the data pat¬ 
tern observed in a previous phase (e.g., a baseline phase). 

The method by which intervention and comparison groups are assigned (group design and 
regression discontinuity design) or the method by which an outcome measure is assessed 
repeatedly within and across different phases that are defined by the presence or absence 
of an intervention (single-case design). Designs eligible for WWC review are randomized 
controlled trials, quasi-experimental designs, regression discontinuity designs, and single¬ 
case designs. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 
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Glossary of Terms 

Eligibility 

Group design 


Intervention 

Intervention report 


Maintenance probes 

Multiple baseline 
design 

Outcome domain 
Phase 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 


Regression 
discontinuity design 
(RDD) 


Reversal/withdrawal 

design 

Single-case design 


A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A study design in which outcomes for a group receiving an intervention are compared to 
those for a group not receiving the intervention. Comparison group designs eligible for 
WWC review are randomized controlled trials and quasi-experimental designs. 

An educational program, product, practice, or policy aimed at improving student outcomes. 

A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an interven¬ 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 

In single-case design research, maintenance probes measure outcomes to understand if 
desired effects are retained after the intervention has ended. 

A single-case design that introduces the intervention at different times to different cases or 
to the same case in different settings. 

A group of closely-related outcomes. A domain is the organizing construct for a set of related 
outcomes through which studies claim effectiveness. 

In single-case design research, phases are the different conditions or varying levels of the 
intervention under which an outcome variable is measured. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

For group design research, the WWC rates the effectiveness of an intervention in each 
domain based on the quality of the research design and the magnitude, statistical signifi¬ 
cance, and consistency in findings. For single-case design research, the WWC rates the 
effectiveness of an intervention in each domain based on the quality of the research design 
and the consistency of demonstrated effects. The criteria for the ratings of effectiveness are 
given in the WWC Rating Criteria on p. 39. 

A design in which groups are created using a continuous scoring rule. For example, stu¬ 
dents may be assigned to a summer school program if they score below a preset point on a 
standardized test, or schools may be awarded a grant based on their score on an applica¬ 
tion. A regression line or curve is estimated for the intervention group and similarly for the 
comparison group, and an effect occurs if there is a discontinuity in the two regression lines 
at the cutoff. 

A single-case design that introduces the intervention and withdraws the intervention. The 
design may be extended by adding additional baseline and/or intervention phases. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 
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Glossary of Terms 


Study rating The result of the WWC assessment of a study. The rating is based on the strength of the 
evidence of the effectiveness of the educational intervention. Studies are given a rating of 
Meets I/I/WC Design Standards Without Reservations , Meets WWC Design Standards With 
Reservations, or Does Not Meet WWC Design Standards, based on the assessment of the 
study against the appropriate design standards. The WWC has design standards for group 
design, single-case design, and regression discontinuity design studies. 

Systematic review A review of existing literature on a topic that is identified and reviewed using explicit meth¬ 
ods. A WWC systematic review has five steps: 1) developing a review protocol; 2) searching 
the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high quality studies and their find¬ 
ings; 4) combining findings within and across studies; and, 5) summarizing the review. 


Visual analysis A visual analysis reviews the pattern of outcome data in a single-case design experiment to 
determine whether the intervention led to a positive (favorable) effect, a negative (unfavor¬ 
able) effect, or no effect is demonstrated. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Intervention 

Report 



Practice 

Guide 



Quick 

Review 


Single Study 
Review 



An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Mathematica Policy Research under contract ED-IES-13-C-0010. 
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